Apparatuses, systems, and methods for confirming identity

ABSTRACT

A system, apparatus, and method for confirming the identity of a person for access using an image of that person. The system includes a camera, a data storage device, a subject identification device, and a processor to compare a plurality of current images of the subject to a plurality of stored images of that subject and confirm that the subject live and an approved subject.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. patent application Ser.No. 14/751,014, filed Jun. 25, 2015, which is incorporated herein in itsentirety and is currently pending.

FIELD OF THE INVENTION

The present invention is concerned with identifying an individual fromone or more images of that individual. Embodiments of the inventionidentify an individual as a live individual depicted in one or moreimages.

BACKGROUND OF THE INVENTION

Computerized recognition of a person from an image of that person's facehas been thought to be beneficial for a variety of biometric securityfunctions. A number of attempts have been made to recognize a person byhis or her facial features, including scanning a face in the infraredspectrum and attempting to identify repetitive movements of the faceduring the scan, infrared scanning and matching of a face with aphotograph, nearest-neighbor matching, Eigenface matching, and Bayesianface recognition.

Different biometric traits have been used in commercialbiometric-related products, including face, fingerprints and iris, witha variety of biometric recognition approaches proposed using one or morebiometric traits, including Principal Component Analysis, LinearDiscriminant Analysis, Local Binary Patterns and fusion techniques atthe image, feature or score level.

In face recognition, the conventional approaches are typically evaluatedin still face images captured under controlled conditions. However,those approaches do not work well when using still images and videoscaptured under unconstrained conditions.

Identifying a person using his or her image, while shown in popularculture, has not yet received general acceptance, possibly because it isunreliable. Thus there is a need for systems, apparatuses, and methodsof identifying a person from the person's image and for confirming thata person is the live person.

SUMMARY OF THE INVENTION

Embodiments of apparatuses, systems, and methods for confirming identityare directed to confirming the identity or liveness of a person or topermitting access to a person approved for such access.

Embodiments of the present apparatuses, systems, and methods forconfirming identity may use a variety of image processing and subjectliveness strategies to improve performance of a system that seeks toidentify a person or determine the liveness of that person through oneor more images.

In accordance with one embodiment of image identification, a live humandetection system includes a camera for capturing a plurality of currentimages of a human subject, a data storage device for storing datacreated from at least a portion of each of the plurality of images ofthe human subject, the portion of each of the plurality of imagesincluding features of at least one eye of the human subject, a subjectidentification device and a processor. In that embodiment, the processorincludes instructions which, when executed by the processor, cause theprocessor to: receive a subject identification from the subjectidentification device; capture a plurality of current images from thecamera, those images including at least one initial image, at least oneintermediate image, and at least one later image, where at least one ofthe plurality of current images can be both an initial image and anintermediate image and at least one other of the plurality of currentimages can be both an intermediate image and a later image; convert eachof the plurality of current images received from the camera to greyscaleif the plurality of current images are in color; detect at least one ofthe eyes of the subject in each of the plurality of current images;determine whether an ocular region including the at least one eye in theat least one initial image matches a corresponding open eye in at leastone of the stored images of the subject; determine whether an ocularregion including the at least one eye in at least one intermediate imagematches a corresponding closed eye in at least one of the stored imagesof the subject; determine whether an ocular region including the atleast one eye in at least one later image matches the corresponding openeye in at least one of the stored images of the subject; and concludethat the subject is live and an approved subject if the ocular regionincluding the at least one eye in the at least one initial image matchesthe corresponding open eye in at least one of the stored images of thesubject, the ocular region including the at least one eye in at leastone intermediate image matches a corresponding closed eye in at leastone of the stored images of the subject, and ocular region including theat least one eye in at least one later image matches the correspondingopen eye in at least one of the stored images of the subject.

In accordance with one embodiment of image identification, a method oflive human detection includes: receiving a subject identification fromthe subject identification device; capturing a plurality of currentimages from a camera, those images including at least one initial image,at least one intermediate image, and at least one later image, where atleast one of the plurality of current images can be both an initialimage and an intermediate image and at least one other of the pluralityof current images can be both an intermediate image and a later image;converting each of the plurality of current images received from thecamera to greyscale if the plurality of current images are in color;detecting at least one of the eyes of the subject in each of theplurality of current images; determining whether the at least one eye inan initial image matches a corresponding open eye in a stored image ofthe subject; determining whether the at least one eye in an intermediateimage matches a corresponding closed eye in a stored image of thesubject; determining whether the at least one eye in a later imagematches the corresponding open eye in the stored image of the subject;and concluding that the subject is live and an approved subject if theat least one eye in the initial image matches the corresponding open eyein the stored image of the subject, the at least one eye in theintermediate image matches the corresponding closed eye in the storedimage of the subject, and the at least one eye in the later imagematches the corresponding open eye in the stored image of the subject.

Accordingly, the present invention provides solutions to theshortcomings of prior apparatuses, systems, and methods for confirmingidentity or liveness of a subject. Those of ordinary skill in the artwill readily appreciate, therefore, that those and other details,features, and advantages of the present invention will become furtherapparent in the following detailed description of the preferredembodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and constitutepart of this specification, include one or more embodiments of theinvention, and together with a general description given above and adetailed description given below, serve to disclose principles ofembodiments of image authentication devices, methods, systems, andnetworks.

FIG. 1 illustrates an embodiment of an identity confirmation system andprocess;

FIG. 2 illustrates an embodiment of a method of providing access basedon identity confirmation;

FIG. 3 illustrates an embodiment of a method of determining liveness ofa subject;

FIG. 4 illustrates an embodiment of a blink detection method;

FIG. 5 illustrates an embodiment of a mobile device for use in identityconfirmation and subject liveness;

FIG. 6 illustrates an embodiment of an identity confirmation and subjectliveness system;

FIG. 7 illustrates another embodiment of an identity confirmation andsubject liveness system; and

FIG. 8 illustrates an embodiment of computing device hardware for use inidentity confirmation or subject liveness detection system.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made to embodiments of apparatuses, systems, andmethods for confirming identity, examples of which are illustrated inthe accompanying drawings. Details, features, and advantages of thoseapparatuses, systems, and methods for confirming identity will becomefurther apparent in the following detailed description of embodimentsthereof. It is to be understood that the figures and descriptionsincluded herein illustrate and describe elements that are of particularrelevance to apparatuses, systems, and methods for confirming identity,while eliminating, for purposes of clarity, other elements found intypical computerized access systems.

Any reference in the specification to “one embodiment,” “a certainembodiment,” or any other reference to an embodiment is intended toindicate that a particular feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment and may be utilized in other embodiments as well. Moreover,the appearances of such terms in various places in the specification arenot necessarily all referring to the same embodiment. References to “or”are furthermore intended as inclusive so “or” may indicate one oranother of the ored terms or more than one ored term.

FIG. 1 illustrates an embodiment of an authentication system and process10 that includes identifying facial features from one or more images 12.An image may be a photograph or a frame of a video or other image. Wherethe image is analog, it may be digitized or converted to digital formfor further processing. The image 12 may furthermore include a humansubject 14.

In that embodiment, a sensor, such as sensors 302, 304, 402, and 502illustrated in FIGS. 5-7, and which may be a type of camera, capturesone or more images 12 when actuated. The one or more images may be takenin visible light or infrared light. An image 12 may furthermore includeas a portion of that image or all of that image an image of a face 16.The image 12 or portion of the image 12 that includes the face 16 may betransmitted to and received by a processor, such as processor 950illustrated in FIG. 8, for processing.

It should be noted that the system and method 10 described herein mayinclude processing the image 12, which includes a human subject 14 andface 16, in (depending on the embodiment) multiple ways (e.g., at 18,20, 22, 23, 24 (including 26 and 28), 40 (including 42 and 44), 50, and52) and each processing step may change the image. Thus, each form ofprocessing may operate on a version of the image 12 that was subject toone or more previous steps of processing. However, for simplicity, wecontinue to refer to the image by 12 (including 14 and 16), recognizingthat the image may not be in original form.

If the one or more images 12 are color images or otherwise not ingreyscale format, the images 12 may be converted to greyscale by theprocessor 950 as is illustrated at 18.

Detection of eyes or the ocular region, which may be referred to hereinas eye detection, in one or more images 12 may be performed by theprocessor 950 as shown at 20 to determine where the eyes are in theimage 12 and, possibly, to determine the location of the face 16 in theimage 12 by scaling from the eyes to a normalized face image size. Thatface image may be a canonical face and that face image may be scaled toa standardized size.

At 22, the processor 950 may photometrically normalize on the face 16for illumination invariance, for example, to eliminate the effects ofillumination variations, such as local shadowing and highlights, whilestill preserving the essential elements of visual appearance for moreefficient face recognition. Photometric normalization 22 of the face 16may precede or follow geometric normalization 23 of the face 16. Atechnique, such as histogram equalization, can be performed, forexample, by dividing the face 16 in the image 12 into a left side and aright side and a distribution of pixel intensity can be determined forthe left side of the face 16, which may be referred to as the left tile,and for the right side of the face, which may be referred to as theright tile. The histogram equalization may be of the contrast-limitedadaptive type, to normalize the pixel intensity distribution in eachportion of the image 12. Histogram equalization may also be performed onthe whole face 16 in the image 12. The equalized whole face image maythen be combined with the equalized left side and ride side of the faceto alleviate illumination variance that may exist during acquisition,for example, because of different environmental conditions existing atthe time of acquisition.

At 23, the processor 950 may geometrically normalize the face 16 foundin the image 12. Geometric normalization 23 of the face 16 may followdetecting the eyes 20 in the face 16. Each eye may be placed in a fixedposition by enlarging or reducing the image 12 or face 16 or byrotating, translating, or scaling the image 12 or face 16, or moving theface 16 left, right, up or down to the extent necessary to place theeyes in the predetermined positions. The image resulting from suchgeometrical normalization 23 of the image 12 or face 16 may be referredto as a geometrically normalized face image.

For example, geometrically normalizing the face 16 may include movingthe face 16 until one of the left eye and the right eye is in a firstpredetermined position in an image field. Scaling the face 16 mayinclude expanding or contracting the face around the first predeterminedlocation until the second of the left eye and the right eye is in asecond predetermined location. It should be recognized that the firstpredetermined position may be a predetermined pixel at which the firsteye is centered and the second predetermined location may be on avertical line of pixels or within a range of pixels.

Features may then be identified or extracted from the geometrically andphotometrically normalized face image 16 at 24. Feature extraction mayyield edges that form lines or other shaped groupings of pixels based onthe subject of the photograph's physiology and the contours andgeometric shape of the subject's face. Feature extraction may includeanisotropic diffusion 26, which may reduce noise in the image 12 andface 16 while preserving facial content and without removing significantparts of the face image content, typically face edges, lines or otherfacial features that are important for the interpretation of the faceimage 16. Feature extraction may produce a family of parameterizedimages where each resulting image is a combination between the originalimage and a filter that depends on the local content of the originalimage. The images resulting from anisotropic diffusion 26 may include aconvolution between the original image 12 and a two-dimensionalisotropic Gaussian filter, where the width of the filter increases withthe parameter. This diffusion process may thus be a linear andspace-invariant transformation of the original image.

The application of anisotropic diffusion 26 may include convolution ofthe original image 12 with a Gaussian filter that increases in width.Facial content may furthermore include facial features such as the nose,eyes, and mouth.

Image noise may include spurious and extraneous information in an imagesuch as the geometrically normalized 22 image 12 that likely did notexist in the subject imaged. For example, one type of image noise may bethe variation of brightness or color information in an image 12 and mayinclude what is sometimes referred to as electronic noise in an image.

Image noise may be removed, for example, by removing pixel segmentshaving less than a predetermined number of contiguous pixels, such as anumber in the range of 10 to 25 contiguous pixels, with theunderstanding that such small segments may not be features of thesubject's face, but may be considered image noise.

Top hat segmentation 28 may be performed on an image 12 such as thegeometrically normalized 22 image 12 in certain embodiments, afteranisotropic diffusion 26 has been performed, to further extract featuresfrom the image 12. Top hat segmentation 28 may be used to extract smallelements and details from the extracted features. For example, a featureextracted using anisotropic diffusion 26 may be further extracted usingtop hat segmentation 28.

The original image 12 may be subtracted from the anisotropicallydiffused image 12 by subtracting the pixel intensity values of theoriginal diffused image 12 from corresponding pixel intensity values ofthe opened image 12. The result of that may be used to segment andextract features from the top hat transformed image.

The top hat segments may also be binarized, for example, to make thesegments white or 1-valued pixels and the non-segment portions of thetop hat segmented image black or 0-valued pixels.

When binarizing 28 the image 12, the pixels may initially be viewed onan intensity scale, each pixel, for example, having an intensity on ascale of 0-255. A threshold may then be selected so that pixels havingan intensity below the threshold are assigned zeros, which may bevisualized as black, and pixels having an intensity above the thresholdare assigned ones, which may be visualized as white, in one embodimentand vice versa in another.

The threshold used during binarization may be set at various levels. Thethreshold may be set low, for example at 85 on a scale of 0-255, so thatmore features that exhibit visual light intensity may be recognized aslight pixels and, for example, assigned one values, than would berecognized as light pixels if the threshold were set higher. Alternatelythe threshold may be set near mid-range, such as at 128 or it may be sethigh in the range, such as around 170 to reduce the number of pixelsrecognized as light and, thereby, reduce artificial or non-featurecarrying pixels or segments made up of contiguous pixels. Any intensityrange may be used as a threshold, including, for example, a thresholdset from 75 to 180.

In a binarized image 12, pixels may be viewed as white and black. Whitetop hat segmented images 12 may be obtained using an anisotropicallydiffused image 12. The anisotropic diffusion of the image 12 may reducenoise in the image without blurring edges of features found in the image12. Then, the image 12 may be opened by dilation of the original image12 followed by erosion of the result. The top hat transformation may beequal to the difference between the intensity of pixels in the imagebeing processed and the result of the open function.

Image noise may include spurious and extraneous information in an imageor processed image, such as the top hat and binarized image 28. Imagenoise may be removed, for example, by removing pixel segments havingless than a predetermined number of pixels, such as a number in therange of 10 to 50, of contiguous pixels with the understanding that suchsmall segments may not be features of the subject's face.

Segments of contiguous pixels that are smaller than a predeterminedthreshold (number) of pixels may be removed from the image 12. Segmentremoval may be used to discard minor features that may not beconsistently detected and noise and to ensure that only prominent andconsistent segmented features are extracted and matched between sampleimages of an individual subject. That may be done to improve the chancethat artificial features and segments, for example those caused by noiseor reflections that do not represent features of the subject, are notconsidered during matching.

A mask, which may be elliptical in shape, may also be applied to thefeature segmented image 24 to ensure that features and segments near theouter borders of the image 24, for example features that are not on theface of the subject or that are not face-based, such as the ears or neckof the subject, may not be considered during matching. The ellipticalmask may be applied after feature extraction so that artificial segmentsare not created and extracted through the appearance of an ellipticalmask in the greyscale image.

Matching of images 12 at 40 may be performed in a variety of ways. Theimages 12 to be matched may include a live image of a person currentlystanding in front of a camera, such as the camera 302 or 304 of a mobiledevice 300 or camera 402, 502, and 902 described herein, an image ofthat person placed in front of the camera 302, 304, 402, 502, or 902 anda stored image of that person. In certain embodiments, all three ofthose images 12 will be matched to assure, for example, that a liveperson requesting access is the same person in an image provided by thatperson at the time of access request and a stored image of the sameperson. One or more of those three images 12 may be (or may have been)subject to processing according to one or more of the processesdescribed above and herein.

The image 12 of the subject that is provided by that person at the timeaccess is requested, where a still image is required for access, may forexample, be a photograph of the person requesting access found in agovernment issued identification document that is held up by the liveperson near the face of that person for reimaging by the camera 302,304, 402, 502, 902. Alternately, the still image 12 may be an image of astill image of the subject taken separately from the image 12 of thelive person. Then, a match between the live person image or images beingtaken currently, the photographic image concurrently provided by thelive person, and an image of a person stored in or accessible to theconfirmation system 10, may be made. If all three of those images aredetermined to be of the same person, and that person is a person who ispermitted access, for example, to a software program that may be run onthe mobile device 300, a room or compartment through a door 412, drawer,or other physical object, as illustrated in FIG. 6, or a financialsystem, as illustrated in FIG. 7, then the requestor is permittedaccess, as requested.

Thus, in an embodiment wherein three images 12 are compared to confirmthe identity of the subject, the subject holds a photograph of thesubject near or next to the face of the subject while an image 12 istaken of both the live subject and the photograph. Then, beforeconfirmation that the live subject is approved for access by comparisonof the live subject to one or more stored gallery images 12, the imageauthentication system 10 confirms that the live subject is the samesubject depicted in the photograph. Thus, the image authenticationsystem 10 may further capture an image 12 of a photograph of the subjectin the same image frame in which the live subject image 12 is capturedusing the mobile device 300 camera 302 or 304 or camera 402 or 502. Theimage authentication system 10 may also cause the processor 950 (asdescribed herein, such as with respect to FIG. 8) to separate the imageof the photograph of the subject from the image of the live subject,convert the photographic image to greyscale if that image is in color,detect the eyes or ocular area of the subject in the photographic image,fix at least one of the left eye or ocular area and the right eye orocular area in a predetermined position in a photographic image field,scale a face in the photographic image to a standard size based on thelocation of the eyes or ocular area in the photographic image 12,equalize lighting on a left side of the face and a right side of theface in the photographic image, binarize the photographic image, groupcontiguous pixels in the binarized photographic image into segments,identify features of the face in the photographic image, remove segmentshaving fewer than a predetermined number of contiguous pixels from thefeatures of the face in the photographic image, mask the photographicimage to remove segments around the face, including ears and neck of thesubject, compare the live image to the photographic image, and determinewhether the live image and the photographic image display the face ofthe same person.

Matching may be performed in the image authentication system and method10 by one or more matching systems. A first matching system may bereferred to as a global matcher 42 and a second matching system may bereferred to as a local matcher 44.

In an embodiment of global matching 42, features that have beenextracted, for example, using feature extraction 24, from a first image12 are compared to features extracted from a second image 12. A score,which may be a ratio of matching pixels to total pixels may bedetermined that indicates the level of any match found between the firstimage 12 and the second image 12. This score or ratio measures theoverlap of extracted segments between the subject image 12 and areference image 12. The score or ratio of the two feature segmentedimages 12 may be computed through convolution. In convolution, the probeimage may be slid pixel by pixel across the gallery image in a top tobottom and left to right fashion, comparing the two images 12 at eachrelative position or in the vicinity of that position, thereby making apixel by pixel match between the images 12.

To account for the small differences that exist in the segmentedfeatures from different samples of the same subject, a predeterminedwindow, such as a 5×5 pixel window, may be formed around a white pixelfor template matching pixels that are not in precisely the samelocation. Thus, for example, if for a white pixel in the probe segmentedimage, there is another white pixel in the 5×5 neighborhood of thecorresponding position on the gallery segmented image 12, then thepixels may be said to be matched. Other sized predetermined windows mayalternately be used, such as, for example, anywhere in the range of a2×2 to a 15×15 pixel window.

In an embodiment of local matching 44, fiducial or reference points inimages 12 may be compared. For example, an end of a feature segment in alive image 12 provided by the camera 302, 304, 402, 502 and from whichfeatures have been extracted at 24 can be compared to a stored referenceimage 12 from which features have been extracted to determine whether asimilar segment exists in the reference image 12 and whether thatsegment ends in a similar location.

Another example of a fiducial or reference point that can be compared infeatures extracted from two images 12 is a point where a segment branchoccurs in a first image 12, a still image 12 presented to the camera302, 304, 402, 502, 902, for example, which can be compared to a secondimage 12, the reference image or live image, for example, to determinewhether a segment branch occurs in or near the same location on thesecond image. Thus, segments that contact or intersect other segmentsmay be given more weight in local matching than segments that do notcontact or intersect other segments. Alternatively, segments thatcontact or intersect other segments may be the only segments consideredin local matching because ridge segments with one or more branchingsegments may be effective to determine whether a local match exists.

To make the local matching algorithm efficient, fiducial points can beremoved post-processing. To define under which conditions fiducialpoints that may be false or otherwise not helpful are to be removed, avariable D, which represents the average inter-segment width or theaverage distance between two parallel neighboring segments, may beintroduced. Four conditions that can be checked in fiducial pointremoval include: (1) if the distance between one branch point and oneend point is less than a predetermined distance D and the two points arelocated in the same segment, remove both points; (2) if the distancebetween two branch points is less than the predetermined distance D andthey are in the same ridge, remove both branch points; (3) if two endpoints are within the predetermined distance D and their directions arecoincident with a small angle variation, and no other termination islocated between the two endpoints, then the two endpoints can beregarded as false fiducial points derived from a broken segment andremoved; and (4) if two end points are located in a short ridge with alength less than the predetermined distance D, the two end points can beremoved.

Fiducial points that are closer to one another than a predeterminednumber of pixels may be eliminated to reduce the computationalcomplexity of the matching step. The fiducial point filtering may, forexample, be done based on the correspondence of points being no morethan 5 to 15, in the following example 10, pixels apart. In such anembodiment, any points extracted from the same segmented features fewerthan 10 pixels apart may be removed, for example beginning with secondpoint in the set or template. The local matcher 150 may also compare thefiducial points in feature extractions from one image 12 to another,after undesired points are eliminated, where such point elimination isused, to determine a number of points that match and a number of pointsthat do not match in those two images 12.

A similar location may be defined as within a predetermined number ofpixels, for example 5-15, from the location found in another image 12.

The global matching result and the local matching result can benormalized so that those results are on the same or comparable scales,for example a 0-1 scale. A fused score may then be determined at 50based on the normalized results of both global matching and localmatching.

The score may be created for one or more of i) comparison of an image 12taken of a live subject and a still image 12 identification photographof that person that is provided by the live subject; ii) comparison ofan image 12 taken of a live subject and a gallery image 12 stored in adatabase in the data storage device 962; and iii) comparison of thestill image 12 identification photograph of that person that is providedby the live subject and a gallery image 12 stored in the database in thedata storage device 962. A matching decision may then be made at 52based on those one or more scores.

In the event that an image 12 is imperfect, the image 12 may besubjected to photometric normalization, image denoising, or imagedeblurring. Image denoising may remove image noise or suppress visualartifacts and image deblurring may remove image blur when the imagecaptured is small or affected by camera motion blur. Photometricnormalization, image denoising, or image deblurring may be performed bythe processor 950 or by a networked device to correct illuminationvariation or other image noise issues and provide a resulting image thatmay be improved and may be utilized for comparison and matching.

To capture better quality images using the camera 302, 304, 402, 502, ahigh frame rate may be selected, image stabilization may be employed, orimage filters, motion stabilizers, or one or more other features builtinto the camera 302, 304, 402, 502 may be employed by the processor 950or another device or person.

When an image 12 of a live subject is to be captured, whether the image12 is a video or one or more still images, a shape may appear on theview finder of the camera 302, 304, 402, 502, 902, which may be thescreen 306 of a mobile device 300 or screen 506, and the subject mayplace his or her face or a desired portion of his or her head, head andshoulders, or any desired portion of his or her person in the shape. Thesubject may attempt to fill the shape with a desired portion of his orher person to standardize the photograph or to ensure capture of thenecessary portion of the subject at a desired image size. Instructionsmay also or alternately be provided on the screen 306 to assist in imagecapture.

The shape appearing on the view finder can be any desired shapeincluding a circle, an oval, or a rectangle.

A standard distance for the subject to be placed from the camera 302,304, 402, 502, 902 may be established and may be communicated to thesubject. For example, where a fixed camera is used, such as the camera402 illustrated in FIG. 6, a line 410 may be placed on the floor at adesired space from the camera 402 and the subject may stand with her orhis toes touching the line 410 or a short distance behind the line 410when the image is captured. Where a mobile device 300 is used, themobile device 300 may be held at arms-length from the subject when theimage 12 is captured.

FIG. 2 illustrates an embodiment of a process and system for enrollingand authenticating a subject 100 in an image authentication system, suchas the image authentication system 10, for example. At 102, a subjectmay select whether they wish to enroll in the image authenticationsystem or be authenticated by the image authentication system. Thatselection may be performed in various ways including, for example, byselecting one of two or more facilities presented on a computer ormobile device screen using, for example, touch screen, a mouse device,or a keyboard.

In one embodiment, this image identification enrollment process 100 maybe open to any subject, while in another embodiment, this imageidentification system enrollment 100 may only be open to one or morepre-registered subjects. In additionally, this image identificationsystem enrollment may be for a new registration system, while in anotherembodiment, this image identification system may be a new component ofany existing registration system.

At 104, the process 100 for enrolling and authenticating a subject in animage authentication system will permit one or more gallery images(e.g., images 12 as described above) to be input into the system 100and, when an appropriate number of gallery images of an acceptablequality have been input, the system 100 will permit the gallery to beset. A subject or user may be provided with a choice of creating agallery or authenticating the subject against a previously input galleryby the system 100 and may select whether to input images or to selectthat subject's gallery. The process for enrolling and authenticating asubject in an image authentication system 100 may require apredetermined number of images be input in a gallery for any subject andmay require that the images input are of a good enough quality to bematched to the subject when that subject requests authentication, forexample, to access a program, application, or location.

At 106, when a selection is made to create a gallery at 104, the subjectmay be instructed by the system 100, for example using text displayed ona computer monitor or mobile device 300 screen 306, or computergenerated audio or voice instructions, for image capture. For example,the subject may be instructed to stand in a predetermined location, tolook at the camera 302, 304, 402, 502, which may, for example, bemounted or may be in a mobile device 300 held by the subject, and thesubject may be instructed to actuate an image capture initiation buttonwhen the image is ready to be captured. The system may process acaptured image at 106, determining whether the image is of a qualityfrom which the subject can be identified, and then display the image tothe subject. The subject may then choose to add the image to thesubject's gallery in the system 100 or may reject the image and have theimage retaken.

At 108, preprocessing of the image takes place. Such preprocessing mayinclude, for example, as described above with respect to theauthentication system and method 10 of FIG. 1, conversion of the imageto greyscale 18, eye detection 20, and geometric normalization 22.Features are extracted at 110 (e.g., as described with respect to method10 at 24) and the extracted features may be stored in a storage device112 for future reference. Points are extracted at 114 (e.g., asdescribed with respect to method 10) and may be stored in a storagedevice 116 for future reference.

At 120, a determination is made as to whether the subject has a galleryof images of himself or herself enrolled in the system 100. If thesubject does not have a gallery enrolled in the system 100, the subjectis requested to enroll a gallery at 124. If the subject does not have agallery enrolled in the system 100, the subject may be returned to 104for enrollment.

If the subject does have a gallery enrolled at 120, then one or moreimages of the subject may be captured at 128 and a determination ofwhether the subject image matches one or more gallery images of thesubject may be performed. That matching may include global matching 130,local matching 132 and fusion 134, which may operate similar to thatdescribed in connection with global matching 42, local matching 44, andfusion 50 discussed elsewhere herein.

FIG. 3 illustrates an embodiment of a liveness detection process 200.That process 200 may include capture of a video of the subject at 202.In one or more frames of that video, which may be images 12 as describedherein, the face of the subject may be localized at 204 and the eyes ofthe subject may be localized at 206 as described herein.

Face localization 204 may detect the face of a subject in a capturedimage 12. Such face localization 204 may be performed using, forexample, an LBP-cascade frontal face function. Face localization 204 mayuse a system, such as the OpenCV detectMultiScale function, to detect ahuman face within the field of view of the camera 302, 304, 402, 502,902 or in an image 12 taken by the camera 302, 304, 402, 502, 902. Facedetection from a video feed or still image 12 may divide an image intodifferent size objects and select an object as likely being face basedfrom its size or another quality in one or more frames of video orimages. The portion of the image 12 containing the object likely to beor contain a face or portion of a face may be referred to as a facesub-image. A rectangle or other shape may then be placed around the facesub-image to provide a boundary for that object.

Eye localization 206 may begin by determining where in an image, whichmay be a bounded image found using face localization 204, the eyes orocular region are located. Eye localization 206 may be performed usingHaar-cascading, for example using the OpenCV Haar-cascading function.Eye localization 206 may return two bounded images, which may bereferred to as a right eye sub-image and a left eye sub-image. The eyesub-images may, for example, be rectangular sections of the facesub-image that are each likely to contain an eye. The face sub-image maythen be shifted or scaled, by expanding it or contracting it, forexample, until the bounded eye sub-image portions of the face sub-imageare placed in predetermined locations. For example, the face sub-imagemay be shifted until the right eye sub-image is centered on a pixel thatis a predetermined number of pixels from the left side of the image anda predetermined number of pixels from the top of the image. Next, theimage may be expanded and contracted around the center pixel used forthe right eye sub-image until the left eye sub-image is a predeterminednumber of pixels from the right side of the image.

At 208, an eye-blinking strategy may be used to confirm the identity orliveness of the subject of the image 12. Such an eye-blinking strategy208 may include comparing one or more of the eyes in at least twodifferent images 12.

Eye blinking or closing may be an element used to determine that asubject being photographed or of which a video is or has recently beentaken is the actual, live subject and not an image of the subject.

An embodiment of an eye blinking detection method 250 is illustrated inFIG. 4. That embodiment may include a subject selecting herself or hisself from a display 306, 406, 506 of one or more subjects approved foraccess 251. Alternatively, another person may select the subject 251 tobe confirmed from the display 306, 406, 506. The display 306, 406, 506may display a picture of the subject, the name of the subject, oranother indicator from which the subject may be identified and thesubject or other person may select the desired subject 251 by touching atouch screen display where the desired subject is indicated, or from akeyboard or mouse by, for example, moving a cursor on the display 306,406, 506 until the desired subject is highlighted and then selecting anenter button or clicking the mouse to select the desired subject.

The subject or another person may then actuate a blink-initiation device252, such as, for example, a picture of a button on a mobile device 300screen 306 or a physical button coupled by wire or wirelessly to animage acquisition device, such as a computing device 300, 404, 504, 910or camera 302, 304, 402, 502, 902. After or around the time that theblink-initiation device is actuated, the subject may blink or close andthen reopen their eyes. The image authentication system may then captureone or more images 12 of the person blinking 254, which may be frames ofvideo, of the subject recognizing that at least the first image capturedshould capture the subject with her or his eyes open, at least onesubsequent intermediate image should capture the subject with his or hereyes closed, and at least the last of the later images captured shouldcapture the subject with her or his eyes open. Those images may each beconsidered image 12 as illustrated in FIG. 1 and may be processed asdiscussed in connection with FIG. 1. For example, one or more of thoseimages 12 may be converted to greyscale 18, eyes or ocular regions maybe detected 20 in the image 12, photometric normalization 22 may beperformed on the image 12, geometric normalization 23 may be performedon the image 12, anisotropic diffusion 26 may be performed on the image12, or top hat and binarization 28 may be performed on the image 12.

Alternatively, the selection of the desired subject may initiate captureof the images 12, immediately or after a time delay of a predeterminedamount of time.

The blink detection method 250 may furthermore provide a countdown tothe time when the blink detection system will capture the blinkingimages. Such countdown may be an audible count, for example with a“three, two, one, blink” emanating from the mobile device 300, thespeaker 408, or another device; displayed on a display 306, 406, or 506;or presented otherwise as desired.

After one or more eyes are detected in an image 12, various portions ofthe image 12 may be processed as described in connection with FIGS. 1,3, and 4. For example, once one or more images are determined to haveone or both eyes in the desired open or closed state, only the facialportion of those images 12 may be processed or only one or both eyeportions of those images 12 may be processed.

An eye-blinking determination function 258 of eye blinking detection 208may determine whether an initial image is of the person claiming to bethe subject with one or both eyes open, whether one or more of theintermediate images 12 is an image 12 of the subject with one or moreeyes closed, and whether a later image is an image of the subject withone or more eyes open. For example, a template based matching functionmay correlate an image 12 of the subject, such as a stored image 12 ofthe subject known to be an image in which the subject has her or hiseyes closed, with an intermediate image 12 taken after the blinkinitiation device is actuated. In particular, one or both of the eyesub-images from the two images (the live intermediate image 12 and thestored image 12, for example) may be compared and if the correlation ishigh, then the processor 950 executing the eye blinking determinationfunction 258 may determine that one or both eyes in the live image 12are, in actuality, closed due to a high level of correlation. If thecorrelation between the two images is low, because, for example, thecoloring, brightness, or other feature of the images, particularly inthe eye sub-images, is different, then a determination may be made thatthe eyes in the live image 12 are closed.

In an embodiment, the subject presenting herself or his self is selectedat 251 and a plurality of images 12 from a video feed are captured andprocessed at 252. The number of frames may be predetermined and image 12capture may continue until the desired number of images have beencaptured. For example, the number of frames of images 12 captured may beselected to be in a range of 7 to 12 in one embodiment.

Once one or both eye portions are extracted from each image 12 framecaptured, if more than one face or more than two eyes are found in animage, the largest eyes or the eyes from the largest face may beselected as the eyes of the subject to be verified.

At 256, one or both of the eyes in the initial blinking video frames maybe correlated with corresponding open eyes in a stored image of thesubject. If the features of a first eye from the initial image 12 aredetermined to match the features from the corresponding open eye in thestored image, then that initial image eye may be determined to be thatof the subject and opened. Similarly, if the features of a second eyefrom the initial image 12 are determined to match the features from thecorresponding open eye in the stored image, then that second initialimage eye may be determined to be that of the subject and opened.

At 258, one or both of the eyes from one or more intermediate images 12captured after the initial images from the blinking video frames may becorrelated with corresponding closed eyes in a stored image of thesubject. If the features of a first eye from the initial image 12 aredetermined to match the features from the corresponding closed eye inthe stored image, then that initial image 12 eye may be determined to bethat of the subject and closed. Similarly, if the features of a secondeye from the initial image are determined to match the features from thecorresponding open eye in the stored image 12, then that second initialimage 12 eye may be determined to be that of the subject and closed.

At 260, one or both of the eyes in the later blinking video frames maybe correlated with corresponding open eyes in a stored image 12 of thesubject as described in connection with 256.

A live determination may be made at 262 based on a desired criterion,such as having both eyes of an initial image 12 frame match both eyesfrom a stored open eye image of the subject, having both eyes of anintermediate image 12 frame match both eyes from a stored closed eyeimage of the subject, and having both eyes of a later image 12 framematch both eyes from a stored open eye image of the subject.Alternatively, only one eye may be required to match in any image 12 oronly one or two of the images may be matched, such as only the closedeye image 12 or the closed eye image 12 and one of the open eye images12.

In a ten frame image 12 video capture, for example, the first and secondimages 12 may be selected for a match determination for open eyes 256,the second through ninth images 12 may be selected for a matchdetermination for closed eyes 258, and the ninth and tenth images 12 maybe selected for a match determination for open eyes 260. Thus, an image12 may be used in both an open eye determination 256 and 260 and aclosed eye determination 258. More than one image 12 may furthermore beconsidered in any or all of 256, 258, and 260 since one image 12considered may have the desired eye position and may be found to match,while another image 12 considered may not have the desired eye positionor may not be found to match. If the eye or eyes of any one of theinitial first and second images 12 are found to match the open eyestored image of the subject, the eye or eyes of any one of theintermediate second through ninth images 12 are found to match theclosed eye stored image of the subject, and the eye or eyes of any oneof the later ninth and tenth images 12 are found to match the open eyestored image of the subject, then it may be concluded that a livesubject match has been found.

The correlation may include a predetermined threshold for similaritybetween the eyes of the images 12 in which a high correlation, one thatrises above the threshold, indicates that the eyes in the blinking image12 are not closed and in which a low correlation, one that is below thethreshold, indicates that the eyes in the blinking image are closed at262.

Many potential causes of low correlation between a stored image 12 ofthe subject with his or her eyes closed and a live image 12 of thesubject exist including, for example, an obstruction, such as a hand,that comes between the subject and the camera 302, 304, 402, 502, whichobstructs imaging of one or both eyes.

Referring again to FIG. 3, at 212, movements, activities, or gesturesother than eye blinking or eye closing may also or alternately be usedto determine whether the person attempting to gain access is live andcurrently being imaged live, as opposed to the person attempting to gainaccess not being live in the current imaged, for example, by presentinga photograph of that person to the system 300, 400, or 500. Suchmovements, activities, or gestures may include, for example, blinkingthe left eye, blinking the right eye, blinking both eyes, sequentiallyor simultaneously, moving the head to the right, left, up, down, or acombination of such head movements, moving the lips, holding one or morefingers in front of the camera 302, 304, 402, 502, 902 near the face,moving closer to or farther from the camera 302, 304, 402, 502, 902,applying pressure to the screen 306 of a mobile device 300 with one ormore fingers, covering the camera 302, 304, 402, 502, 902, for examplewith a hand such that the face is not in the field of view of the camera302, 304, 402, 502, 902, speaking, for example saying a phrase that maybe provided by a mobile device 300 by, for example, providing the phrasein writing or audibly, requiring one or more activities, such asconnecting a sequence of dots, circles, disks, or other objects, orusing one or more fingers to touch locations where dots or other objectsappear on the screen, where the appearance of such dots or other objectsmay be randomized.

When images 12 are taken of a subject moving, acting, or gesturing,functions described herein can be performed, including for example,inclusion of a shape in the viewfinder of the camera 302, 304, 402, 502,902 or on a monitor 306, 406, or 506 in which the image 12 to becaptured is displayed, for positioning of the face or other portion ofthe body; use of a still photograph held or otherwise placed in thefield of view of the camera 302, 304, 402, 502, 902, inclusion of texton a screen, such as the screen 964 of a mobile device 306, facelocalization, and eye localization.

When sensing or determining when a movement, activity, or gesture hasoccurred, image authentication may compare two or more images 12. In oneembodiment, an image authentication system 910 has images 12 of asubject stored in a storage device 962 showing the subject making avariety of movements, activities or gestures and compares one or morecurrent images 12 to one or more stored images 12 of the subject makingthe desired gesture or action. The image authentication system 910 maythen determine whether two or more images 12 contain the same subject inthe same position.

In another embodiment, the image authentication system compares one ormore current images 12 to one or more images 12 of the subject notmaking the movement.

Machine-machine detection may be performed on one or more images 12 or avideo of a subject. Machine-machine detection may assist in thwarting anattempt at faking or hacking the liveness required for access in certainembodiments. For example, someone other than the proper subject mayattempt to gain access by, for example, playing a video of the subjectperforming the required action, gesture or movement. To thwart anattempt at access by someone other than the live subject purported to beattempting access, image authentication may attempt to detect screenreflection from a screen 306 being used to display the subject or thesubject performing an action, gesture, or movement requested of thesubject to show liveness at 214. Other ways of detecting that a machineis being used in an attempt to fool the system into believing the livesubject is being imaged when the live subject is not being imagedinclude, detection of an object having the shape of a mobile device 300that might be used to display the action, gesture, or movement requestedof a subject; detection of fingers surrounding the subject performingthe action, gesture, or movement requested of a subject; or backgrounddetection at 216.

Apparatuses, systems, and methods for confirming that a subject isappearing live in the images may detect a reflection that might becreated by a device screen 306, 406, 506 used to display one or moreimages or a video of the subject that the image authentication system orprocess is attempting to confirm is the person requesting access. In onesuch embodiment, image detection may compare movement of the presentedsubject in two or more images 12 and may compare corresponding locationof the pixels impacted by the reflection to ascertain the likelihoodthat the reflection is caused by a machine presentation as opposed to alive presentation of the subject.

Two or more different image thresholding techniques can be used toconvert the greyscale image to different resulting images in anembodiment of reflection detection. One of those thresholding techniquesmay be used to find features of the subject at 24 and another of thosethresholding techniques may be used to find reflection at 214 in anembodiment. The predefined intensity level used to create one greyscaleconversion for reflection detection 214 can be greater than thepredefined level used for feature identification so that reflection thatindicates the subject is not live can be detected at 214 and possiblylocated in the image. For example, the reflection detection predefinedintensity level may be greater than 230.

Reflection detection 214 may determine whether the number or grouping ofbright pixels (over the predefined level) in the greyscale conversioncreated for reflection detection 214 indicates that a reflection existsin the live image 12 that may be caused by a screen or other undesirablecondition that exists in the image. For example, in an embodiment, acontiguous grouping of bright pixels that exceeds 10% of the totalpixels in the image indicates that an undesirable reflection that may,for example, be caused by a screen, exist in the image and result in adetermination that a reflection is detected at 214. In anotherembodiment, the location of the group of bright pixels may be used as anindicator that there is an undesirable reflection. For example, if thepixel grouping indicating a reflection exists primarily over the face ofthe subject that might be determined to be an undesirable reflection.Alternately, if the if the pixel grouping indicating a reflection existsprimarily in the background of the face of the subject, that mightindicate an undesirable reflection. In yet another example, a pixelgrouping indicating a reflection that exists partially over the face ofthe subject and the partially over the background around the face in theimage 12 might indicate an undesirable reflection.

In another embodiment, the intensity level found in the reflectiondetection greyscale conversion may be compared to the intensity levelfound in another image, which may be a stored image, for example, todetermine whether indicates that a reflection exists. A template basedmatching function may correlate the live image 12 of the subject withthe other image of the subject. Such a template based matching functionmay compare a stored face image against a live image 12 and, if thecorrelation between those images is low, because, for example, thecoloring, brightness, or other feature of the images, particularly inthe face image, is different, then the processor 950 executing thescreen reflection detection routine at 214 may conclude that anundesirable reflection or reflections exist in the live image.

In yet another embodiment, the intensity level found in two or moreimages captured live, for example at the time that access is beingrequested, may be compared to detect reflection at 214. For example,where a reflection is detected in two or more images 12 in the same or asimilar location, such detection may indicate that the reflection isfrom a screen and is undesirable such that access to that subject wouldbe denied. Alternately, where a reflection is detected in two or moreimages 12 and that reflection is in a different location in each ofthose images 12, such detection may indicate that the reflection is froma screen and is undesirable. Other live detection, such as requiring thesubject to move location and capture another image 12, determination ofwhether a screen shaped or finger shaped object appears in multipleimages 12 at 216, or a determination of face background consistency at218 may be used to assist in making the determination of whether theimage 12 includes the live subject. It should be noted that use ofcertain live determination processes (such as 216 and 218) may beconditioned on a potential issue being detected in a prior process (suchas 214 or, in the case of process 218, 216).

If nothing that appears to be associated with an attempt to fool theprocess 10 or system 910 is discovered, access may be permitted to thesubject at 220 and if there is any apparent machine-machine or otherissue that may be associated with an attempt to fool the process 10 orsystem 910, then access may be denied at 230.

The correlation between images may include a predetermined threshold forthe brightness or color found in those images, below which it may bedetermined that a reflection exists. The correlation may furthermore bebetween the image 12 or the face of the subject in the image 12 and anaverage of more than one stored image 12 or face image.

The image 12, as used throughout this document, may furthermore be anaverage of two or more images. For example, the image 12 may be anaverage of several frames of video taken of the subject live, which maybe averaged on a pixel by pixel brightness basis or another desiredbasis.

Apparatuses, systems, and methods for confirming that a subject isappearing live in the images may detect a shape that might indicate adevice screen is being used to display one or more images or a video ofthe subject that the image authentication system or process isattempting to confirm is the person requesting access. For example, inone embodiment, image detection may determine whether a shape thatindicates presence of a screen is present in one or more images. Suchshapes could include, for example, straight lines or other shapes commonto a machine that could display an image. Such shapes may furthermore befound in the segments of a segmented image. For example, in oneembodiment, two or more straight lines that are directed to anintersection at nearly a 90 degree angle and appear in image space thatsurrounds the face of the subject indicate the possible use of a devicescreen to display the subject. The system may furthermore determinewhether such a shape exists in two or more images, with more than oneimage displaying the shape indicating that a device screen may be in useto display the image.

Apparatuses, systems, and methods for confirming that a subject isappearing live in the images may detect a shape that indicates one ormore fingers or portions of fingers are included in an image around adevice screen used to display one or more images or a video of thesubject for which the image authentication system or process isattempting to confirm is the person requesting access. For example,image detection may determine whether one or more shapes are found inthe segments that are the shape of fingers are found adjacent one ormore segments that could be screen edges, which could indicate it islikely that the presented image is created by a machine presentation andis not a live presentation of the subject.

Apparatuses, systems, and methods for confirming that a subject isappearing live in the images may also use the background behind oraround the subject 218 to detect liveness. In background detection 218,the subject may be requested to perform an action, gesture or movementand then requested to perform the same action, gesture or movement infront of a different background. Image authentication may then determinewhether the background behind the face or body of the subject changed,as it should if the subject is live in front of the camera, or whetherthe background behind the face or body of the subject remains the samein both the first and second actions, gestures, or movements, whichwould indicate that the subject is not live but is rather on arecording. For example, in one embodiment, a system might determinewhether there is a change in the background when the subject is asked tomove to another position for a second image capture. Such a change mightinclude detection of a color, pattern, or another element, which mightbe sensed by, for example, one or more segments in an image or a changein the intensity of pixels in the background of the face by more than apredetermined amount. In such an embodiment, a lack of change in thebackground, after requesting that the subject move, for example, tostand in front of a different wall having a different color or patternthat the wall behind the subject in the first image or set of imagescaptured, may indicate that the image is not a live rendering of thelive subject, but is rather provided by a machine.

FIG. 5 illustrates a mobile device 300 that may be used in embodimentsof image authentication, such as the systems and methods 10 and 100 andotherwise as described herein. The mobile device 300 illustratedincludes a front camera 302, a rear camera 304, a screen 306, controlbuttons 308, and a flash 310. The mobile device 300 may also containcomputing devices as described herein or otherwise included in suchmobile devices 300, including a processor 950 or microprocessor andmemory 962 and may have stored thereon various software applications.The mobile device may furthermore be handheld, as is illustrated in FIG.5, or it may be formed in a wrist bound form, on glasses, or otherwiseas desired. Thus, in one embodiment, image authentication software maybe installed on the mobile device and may operate by requesting that theowner of the mobile device 300 provide a gallery of images of thatowner. To capture an image of the owner, the owner may select whetherone or more images (e.g. image 12 described herein) are to be acquiredusing the front camera 302 or the back camera 304. The owner may thenhold the mobile device 300 at arms-length or another desired distancefrom the owner with the selected camera 302, 304 directed at the owner.The screen 306 may act as a view finder at this time and may show theimage to be captured. The screen 306 may also include the outline of ashape. The owner may then move his or her arm, hand, or body to placehis or her face in the center of the screen 306, possibly within theshape and extending to or nearly to the shape but not extending beyondor far beyond the shape. The owner may then actuate the camera 302, 304so that the image shown on the screen is captured and stored in thememory of the mobile device 300. Such image capture may be performedboth to create a gallery of images of the owner and to authenticate theowner as described herein.

Once a gallery of images of the owner have been taken and are stored inthe mobile device 300, the image authentication system can be used topermit only the owner to access the mobile device 300 or any desiredprograms or functions of the mobile device 300. For example, if theowner wishes to make photographs stored on the mobile device 300accessible only to the owner, the owner can establish the imageauthentication system as a gatekeeper for photographs on the mobiledevice 300. Then, when access to the photographs on the mobile device300 is desired, the subject initiates the photograph storage aspect ofthe mobile device 300 and the mobile device will request that therequestor capture one or more images of the owner or other user havingapproved access using the mobile device 300 camera 302, 304. The imageauthentication system may further ask the requesting subject to move,act, or gesture. The image authentication system then determines whetherthe current subject matches the subject in the gallery. If the imageauthentication system determines the current subject does match thesubject in the gallery then access is permitted to the photographs onthe mobile device 300. If the image authentication system does notdetermine the current subject matches the subject in the gallery thenaccess to the photographs on the mobile device 300 is denied.

FIG. 6 illustrates an embodiment of an image authentication system 400to permit access. The image authentication system 400 illustratedincludes a camera 402, a computing device 404 which may have any or allof the components illustrated in and discussed in connection with FIG. 8or elsewhere herein, a screen 406, a speaker 408, a microphone 418, aposition indicator 410, a door 412, and a lock for a bolt, magneticcloser, latch, or other locking mechanism 414.

In one embodiment, the image authentication system 400 may includesoftware that is installed on the computing device 404 to limit accessto the door 412. The image authentication system 400 may operate byrequesting that a person presenting himself or herself for accessthrough the door 412 further present himself or herself for approval foraccess.

The image authentication system 400 may be used to capture a gallery ofimages for each of a plurality of individuals who are approved foraccess. Those galleries may be stored in a data storage device containedin or otherwise associated with the computing device 404, associated forexample by a local bus or over a wider network that may include theInternet.

When a person's gallery is stored in the image authentication system400, an administrator of the image authentication system 400 may approvethat person for access. That approved person may then present himself orherself and request access. The image authentication system 400 may thenrequest that the person presented position himself or herself for image12 acquisition to gain access approval. For example, the imageauthentication system 400 may request that the person stand with his orher toes on a position indicator 410 that is a line as shown in FIG. 6,facing the camera 402. The line or another position indicator 410 mayassist with placing the subject at a location and distance from thecamera 402 that enables the capture of an image of that subject. Theimage authentication system 400 may present requests to the subject inany way desired, including by voice commands provided from the computer404 through the speaker 408 or through text or images displayed to thesubject on the screen 406.

Once the subject requestor is positioned for image capture, the subjectmay inform the image authentication system 400 that an image should becaptured by, for example, speaking a command or pushing a button, whichmay be associated with or appear on the screen 406. The camera may thenbe actuated to capture an image (e.g., image 12) of the subject.

Next, the image authentication system 400 may process the image capturedand compare that image or the resultant data from the processed image tothe gallery of images for that person or the processed data taken fromthose images. Such processing may be as described herein, such as withrespect to the image 12 above and herein. The image authenticationsystem 400 may further ask the requesting subject to move, act, orgesture. The image authentication system 400 then may determine whetherthe current subject is live or matches the subject in the gallery. Ifthe image authentication system 400 determines the current subject doesmatch the subject in the gallery and, if applicable, is live, then anactuator for the locking mechanism 414 is actuated to permit the door412 to open so that access is permitted to the subject through the door412. If the image authentication system 400 does not determine thecurrent subject matches the subject in the gallery or is not live thenaccess through the door 412 may be denied.

FIG. 7 illustrates an embodiment of an image authentication andinformation provision system 500 to confirm the identity of a subjectand provide information related to that subject. For example, the imageauthentication system 500 may be used at an immigration checkpoint orother government counter to provide an immigration officer or otheremployee with pertinent information about the person presenting himselfor herself; at a bank or other service counter to provide a serviceprovider with information such as account information for a customer; orat a retail outlet where a customer loyalty card, credit card, oranother form of identification may be used to confirm the identity ofthe person presenting herself or himself and provide the cashier orother employee with information that may make the customer's experiencebetter or may help the retailer suggest additional purchases. The imageauthentication and information provision system 500 can include anydevice or function described herein and, as illustrated, includes acamera 502, a computing device 504 which may have any or all of thecomponents illustrated in and discussed in connection with FIG. 8 orelsewhere herein, a screen 506, and a printer 508.

In one embodiment, the image authentication and information provisionsystem 500 includes software that is installed on the computing device504 to confirm the identity of the subject presenting herself or himselfas described herein and may provide pertinent information regarding thatperson to a person to whom the subject is presenting himself or herself.The image authentication and information provision system 500 mayoperate by having an employee to whom the subject is presented himselfor herself request that the subject place herself or himself and anidentification of that person, which may be a passport, a customerloyalty card, or other identification that includes a photograph, infront of the camera 502.

The image authentication and information provision system 500 mayprocess the image captured and compare that image or the resultant datafrom the processed image to the photograph in the identification. Suchprocessing may be as described herein, such as with respect to the image12 above and herein. The image authentication and information provisionsystem 500 may further ask the requesting subject to move, act, orgesture. The image authentication and information provision system 500then may determine whether the current subject is a live subject ormatches the subject in the identification. If the image authenticationand information provision system 500 determines the current subject doesmatch the subject in the gallery then an affirmative match may bedisplayed on the screen 506 and pertinent information regarding thatperson may also be displayed on the screen 506. If the imageauthentication and information provision system 500 does not determinethe current subject matches the subject in the identification then thelack of a confirmed match may be displayed on the screen 506.

FIG. 8 illustrates computing device hardware 910 that may be used in anembodiment of an image authentication system, such as one of the imageauthentication systems 10, 100, 400 described herein. As shown in FIG.8, the computing device 910 may include a processor 950, memory 952, astorage device 962, a coupling for an output device 964, a coupling foran input device 966, a communication adaptor 968, and a camera 902.Communication between the processor 950, the storage device 962, theoutput device coupling 964, the input device coupling 966, and thecommunication adaptor 968 is accomplished by way of one or morecommunication busses 970. It should be recognized that the controlcircuitry 910 may have fewer components or more components than shown inFIG. 8. For example, if a user interface is not desired, the inputdevice coupling 966 or output device coupling 964 may not be includedwith the control circuitry 910.

The memory 952 may, for example, include random access memory (RAM),dynamic RAM, or read only memory (ROM) (e.g., programmable ROM, erasableprogrammable ROM, or electronically erasable programmable ROM) and maystore computer program instructions and information. The memory 952 mayfurthermore be partitioned into sections including an operating systempartition 958 where system operating instructions are stored, and a datapartition 956 in which data is stored.

The processor 950 may be any desired processor or microprocessor,including a processor in a mobile device or tablet or a processor in ageneral purpose computer or server. The processor 950 may, for example,be an Intel® type processor or another processor manufactured by, forexample AMD®, DEC®, or Oracle®. The processor 950 may furthermoreexecute the program instructions and process the data stored in thememory 952. In one embodiment, the instructions are stored in memory 952in a compressed or encrypted format. As used herein the phrase,“executed by a processor” is intended to encompass instructions storedin a compressed or encrypted format, as well as instructions that may becompiled or installed by an installer before being executed by theprocessor 950.

The storage device 962 may, for example, be non-volatile battery backedSRAM, a magnetic disk (e.g., hard drive), optical disk (e.g., CD-ROM) orany other device or signal that can store digital information. Thecommunication adaptor 968 permits communication between the computer 910and other devices or nodes coupled to the communication adaptor 968 atthe communication adaptor port 972. The communication adaptor 968 may bea network interface that transfers information from a node such as ageneral purpose computer to the computing device 910 or from thecomputing device 910 to a node. It will be recognized that the computingdevice 910 may alternately or in addition be coupled directly to one ormore other devices through one or more input/output adaptors (notshown).

The input device coupling 966 and output device coupling 964 may coupleone or more input or output devices. It will be recognized, however,that the computing device 910 does not necessarily need to have an inputdevice or an output device to operate. Moreover, the storage device 962may also not be necessary for operation of the computer 910 as data maybe stored in memory, for example. Data may also be stored remotely andaccessed over a network, such as the Internet.

The elements 950, 952, 962, 964, 966, and 968 related to the computingdevice 910 may communicate by way of one or more communication busses970. Those busses 970 may include, for example, a system bus or aperipheral component interface bus.

The camera 902 may be a variety of image acquisition devices andsensors, including, for example, a digital camera including a digitalsingle-lens reflex (DSLR) camera, a camera in a mobile device such as anAndroid device or iPhone, a thermal camera, a photonic capture device,or another device that can capture an image and transfer that image to aprocessor 950 or storage device 952. The camera 902 may, in variousembodiments, be any of the cameras described herein. The imageauthentication system described in one or more of the embodiments hereinmay select an image or video acquisition available on the device,adjusting, for example, aperture, lens length, shutter speed, and whitebalance on a DSLR camera or the high dynamic range setting, flash, andsquare format on an iOS device. For example, if the device on whichimage authentication is taking place is an iPhone, the imageauthentication system may select to have a video of the subject taken athigh resolution with high dynamic range setting in square mode and maysense the light present on the subject to determine whether to turn onthe flash. In one embodiment, such settings would be predetermined bythe system configuration, while, in other embodiments, settings could bedetermined by a system user or by device configuration settings. In ahigher quality camera, the image authentication system may select aresolution that is adequate for image authentication, but is not such ahigh resolution to cause issues, such as capture speed. For example,with some higher resolution cameras, full resolution images could causeundesirable delays in capture speed, image processing speeds, or imagematching speed. The camera 902 may capture a visual image of the face ofa person toward which the camera 902 is directed. The camera 902 may beany camera including a camera that is a part of a computer device, suchas a mobile device or a tablet computing device, or may be a camera thatcaptures an image digitally and transmits that image to a computingdevice.

While the present invention has been disclosed with reference to certainembodiments, numerous modifications, alterations, and changes to thedescribed embodiments are possible without departing from the scope ofthe present invention, as defined in the appended claims. Accordingly,it is intended that the present invention not be limited to thedescribed embodiments, but that it have the full scope defined by thelanguage of the following claims, and equivalents thereof.

What is claimed is:
 1. A live human detection system, comprising: acamera for capturing a plurality of current images of a human subject; adata storage device for storing a plurality of stored images of thehuman subject, each stored image including at least one of features ofat least one eye of the human subject and data created from an image ofat least one eye of the human subject; a subject identification device;and a processor coupled to the data storage device, the camera, and thesubject identification device, the processor including instructionswhich, when executed by the processor, cause the processor to: receive asubject identification from the subject identification device; capture aplurality of current images from the camera, those images including atleast one initial image, at least one intermediate image, and at leastone later image, where at least one of the plurality of current imagescan be both an initial image and an intermediate image and at least oneother of the plurality of current images can be both an intermediateimage and a later image; convert each of the plurality of current imagesreceived from the camera to greyscale if the plurality of current imagesare in color; detect at least one of the eyes of the subject in each ofthe plurality of current images; determine whether an ocular regionincluding one of the detected eyes in the at least one initial imagematches a corresponding open eye in at least one of the stored images ofthe subject; determine whether an ocular region including one of thedetected eyes in at least one intermediate image matches a correspondingclosed eye in at least one of the stored images of the subject;determine whether an ocular region including one of the detected eyes inat least one later image matches the corresponding open eye in at leastone of the stored images of the subject; and conclude that the subjectis live and an approved subject if the ocular region including the atleast one eye in the at least one initial image matches thecorresponding open eye in at least one of the stored images of thesubject, the ocular region including the at least one eye in at leastone intermediate image matches a corresponding closed eye in at leastone of the stored images of the subject, and ocular region including theat least one eye in at least one later image matches the correspondingopen eye in at least one of the stored images of the subject.
 2. Thelive human detection system of claim 1, further comprising: an inputdevice coupled to the processor; and a display coupled to the processor;and wherein the processor further includes instructions which, whenexecuted by the processor, cause the processor to: display one or moreapproved subjects on the display; receive an input from the input devicethat indicates which displayed approved subject is the human subjectpresented; and retrieve the stored image of the selected approvedsubject with one or more eyes closed and the stored image of theselected approved subject with one or more eyes open.
 3. The live humandetection system of claim 1, wherein the processor further includesinstructions which, when executed by the processor, cause the processorto: binarize each of the plurality of current images; group contiguouspixels in each of the binarized plurality of current image intosegments; identify features of the ocular region including one of thedetected eyes in each of the plurality of current images; removesegments having fewer than a predetermined number of contiguous pixelsfrom the features of the ocular region including one of the detectedeyes; mask each of the plurality of current images to remove segmentsaround the ocular region including one of the detected eyes.
 4. Thesystem of claim 3, wherein the grouping of the contiguous pixelsincludes top hat segmenting the current image.
 5. The live humandetection system of claim 1, further comprising: a blink-initiationdevice coupled to the processor; and wherein the processor furtherincludes instructions which, when executed by the processor, cause theprocessor to: receive a blink-initiation signal from the blinkinitiation device; and receive the plurality of current images from thecamera after receiving the blink-initiation signal.
 6. The live humandetection system of claim 5, further comprising: a speaker coupled tothe processor; and wherein the processor further includes instructionswhich, when executed by the processor, cause the processor to: providean audible countdown through the speaker; and capture the plurality ofimages at at least one of during the countdown and after the countdown.7. The live human detection system of claim 1, wherein the subjectidentification device is a touchscreen display for displaying list ofapproved subjects and on which the subject to be confirmed is to beselected from the list.
 8. The live human detection system of claim 7,wherein the at least one later image is at least one of the last of theplurality of current images captured and at least one image capturedconsecutively before the last of the plurality of current imagescaptured.
 9. The live human detection system of claim 8, wherein the atleast one initial image is at least one of the first of the plurality ofcurrent images captured and at least one image captured consecutivelyafter the first of the plurality of current images captured.
 10. Thelive human detection system of claim 1, wherein the at least one laterimage includes the last of the plurality of current images captured. 11.The live human detection system of claim 1, wherein the at least oneinitial image includes the first of the plurality of current imagescaptured.
 12. The live human detection system of claim 1, wherein the atleast one intermediate image is captured after the first of theplurality of current images is captured and before the last of theplurality of current images is captured.
 13. The live human detectionsystem of claim 1, wherein the processor further includes instructionswhich, when executed by the processor, cause the processor to mask theocular region surrounding and including the at least one eye afterdetecting the at least one eye of the subject in each of the pluralityof images.
 14. The live human detection system of claim 1, wherein theprocessor further includes instructions which, when executed by theprocessor, cause the processor to: detect both eyes of the subject ineach of the plurality of current images; determine whether both eyes ofthe subject in an initial image match both corresponding open eyes in astored image of the subject; determine whether both eyes in anintermediate image match both corresponding closed eyes in a storedimage of the subject; determine whether both eyes in a later image matchboth corresponding open eyes in the stored image of the subject;conclude that the subject is live and an approved subject if both eyesin the initial image match both corresponding open eyes in the storedimage of the subject, both eyes in the intermediate image match bothcorresponding closed eyes in the stored image of the subject, and botheyes in the later image match both corresponding open eyes in the storedimage of the subject.
 15. The live human detection system of claim 14,wherein the function is an application.
 16. The live human detectionsystem of claim 1, wherein the processor further includes instructionswhich, when executed by the processor, cause the processor to access afunction of a computing device.
 17. The live human detection system ofclaim 1, further comprising: determining a score based on the degreethat the at least one initial image matches at least one of the storedimages, the at least one intermediate image matches at least one of thestored images, and the at least one later image matches at least one ofthe stored images; and wherein concluding that the subject is live andan approved subject further includes determining that the score isgreater than a predetermined score.
 18. The live human detection systemof claim 1, wherein the binarizing of the current image includesdetermining the intensity of each pixel in the current image and causingall pixels having an intensity below a predetermined threshold to beblack and causing all pixels having an intensity equal to or above thepredetermined threshold to be white.
 19. The live human detection systemof claim 1, wherein: the subject is further required to perform one ofblink only one eye and blink both eyes sequentially; both eyes of thesubject are detected in each of the plurality of current images; andconcluding that the subject is live requires that the processordetermine that both eyes in the at least one initial image match atleast one of the stored images, that both eyes in the at least oneintermediate image match at least one of the stored images, and thatboth eyes in the at least one later image match at least one of thestored images.
 20. A method of live human detection, comprising:receiving a subject identification from the subject identificationdevice; capturing a plurality of current images from a camera, thoseimages including at least one initial image, at least one intermediateimage, and at least one later image, where at least one of the pluralityof current images can be both an initial image and an intermediate imageand at least one other of the plurality of current images can be both anintermediate image and a later image; converting each of the pluralityof current images received from the camera to greyscale if the pluralityof current images are in color; detecting at least one of the eyes ofthe subject in each of the plurality of current images; determiningwhether the at least one eye in an initial image matches a correspondingopen eye in a stored image of the subject; determining whether the atleast one eye in an intermediate image matches a corresponding closedeye in a stored image of the subject; determining whether the at leastone eye in a later image matches the corresponding open eye in thestored image of the subject; and concluding that the subject is live andan approved subject if the at least one eye in the initial image matchesthe corresponding open eye in the stored image of the subject, the atleast one eye in the intermediate image matches the corresponding closedeye in the stored image of the subject, and the at least one eye in thelater image matches the corresponding open eye in the stored image ofthe subject.